System and method for applying Link Analysis Tools for Visualizing Connected Temporal and Spatial Information on a User Interface

ABSTRACT

A system and method for analyzing a plurality of data elements having both temporal and spatial properties, where a first data element and a second data element of the plurality of data elements are linked by at least one association element. The system and method include selecting the first data element from the plurality of data elements and providing at least one search criteria for use in analyzing the properties of the plurality of data elements with respect to at least one property of the first data element. An analysis module is used to apply the at least one search criteria to the properties of the plurality of data elements for identifying the second data element from the plurality of data elements and the corresponding at least one association element. The at least one association element is configured for representing a connection between the first data element and the second data element, such that the connection has a first property common to a property of the first data element and a second property common to a property of the second data element. A visualization module is used to generate a visual representation of the first and second data elements and the association element configured for display on a user interface for subsequent interaction with user events. The visual representation includes a spatial domain including a reference surface for providing a spatial reference frame having at least two spatial dimensions and a temporal domain operatively coupled to the spatial domain for providing a common temporal reference frame for locations of interest in the spatial domain.

(The present application is a Continuation of U.S. Continuation in Partapplication Ser. No. 11/503,921, filed Aug. 15, 2006, which claims thebenefit of U.S. Pat. No. 7,180,516, filed Mar. 29, 2004 and U.S. Pat.No. 7,499,046, filed Mar. 14, 2005, which claims the benefit of U.S.Provisional Application No. 60/707,942, filed Aug. 15, 2005, all hereinincorporated by reference.)

BACKGROUND OF THE INVENTION

The present invention relates to an interactive visual presentation ofmultidimensional data on a user interface.

Tracking and analyzing entities and streams of events, has traditionallybeen the domain of investigators, whether that be national intelligenceanalysts, police services or military intelligence. Business users alsoanalyze events in time and location to better understand phenomenon suchas customer behavior or transportation patterns. As data about eventsand objects become more commonly available, analyzing and understandingof interrelated temporal and spatial information is increasingly aconcern for military commanders, intelligence analysts and businessanalysts. Localized cultures, characters, organizations and theirbehaviors play an important part in planning and mission execution. Insituations of asymmetric warfare and peacekeeping, tracking relativelysmall and seemingly unconnected events over time becomes a means fortracking enemy behavior. For business applications, tracking ofproduction process characteristics can be a means for improving plantoperations. A generalized method to capture and visualize thisinformation over time for use by business and military applications,among others, is needed.

Many visualization techniques and products for analyzing complex eventinteractions only display information along a single dimension,typically one of time, geography or a network connectivity diagram. Eachof these types of visualizations is common and well understood. Forexample a Time-focused scheduling chart such as Microsoft (MS) Projectdisplays various project events over the single dimension of time, and aGeographic Information System (GIS) product, such as MS MapPoint, orESRI ArcView, is good for showing events in the single dimension oflocations on a map. There are also link analysis tools, such as Netmap(www.netmapanalytics.com) or Visual Analytics (www.visualanalytics.com)that display events as a network diagram, or graph, of objects andconnections between objects. Some of these systems are capable of usinganimation to display another dimension, typically time. Time is playedback, or scrolled, and the related spatial image display changes toreflect the state of information at a moment in time. However thistechnique relies on limited human short term memory to track and thenretain temporal changes and patterns in the spatial domain. Anothervisualization technique called “small multiples” uses repeated frames ofa condition or chart, each capturing an increment moment in time, muchlike looking at sequence of frames from a film laid side by side. Eachimage must be interpreted separately, and side-by-side comparisons made,to detect differences. This technique is expensive in terms of visualspace since an image must be generated for each moment of interest,which can be problematic when trying to simultaneously display multipleimages of adequate size that contain complex data content.

A technique has been developed, as described in InteractiveVisualization of Spatiotemporal Patterns using Spirals on a GeographicalMap—by Hewagamage et al. that uses spiral shaped ribbons as timelines toshow isolated sequences of events that have occurred at discretelocations on a geographical map. This technique is limited because ituses spiral timelines exclusively to show the periodic quality ofcertain types of events, while does not show connectivity between thetemporal and spatial information of data objects at multi-locationswithin the spatial domain. Further, event data objects placed on thespirals can suffer from occlusion, thereby providing for only a limitednumber of events and locations viewable with the spiral timelines.

Further, there exists problems in simplifying or otherwise analyzingcluttered visualizations with respect to identifying relevant dataelements associated with one another from non-associated data elements.Further, problems exist in changing the content of a visualization toassist the analyst in interpretation of identified data elements.

It is an object of the present invention to provide a system and methodfor the integrated, interactive visual representation of a plurality ofdata elements with spatial and temporal properties to obviate ormitigate at least some of the above-mentioned disadvantages.

SUMMARY OF THE INVENTION

Tracking and analyzing entities and streams of events, has traditionallybeen the domain of investigators, whether that be national intelligenceanalysts, police services or military intelligence. Business users alsoanalyze events in time and location to better understand phenomenon suchas customer behavior or transportation patterns. As data about eventsand objects become more commonly available, analyzing and understandingof interrelated temporal and spatial information is increasingly aconcern for military commanders, intelligence analysts and businessanalysts.

There exists problems in simplifying or otherwise analyzing clutteredvisualizations with respect to identifying relevant data elementsassociated with one another from non-associated data elements, inparticular in environments with both temporal and spatial properties.Further, problems exist in changing the content of a visualization toassist the analyst in interpretation of identified data elements.Contrary to present visualization systems there is provided a system andmethod for analyzing a plurality of data elements having both temporaland spatial properties, where a first data element and a second dataelement of the plurality of data elements are linked by at least oneassociation element. The system and method include selecting the firstdata element from the plurality of data elements and providing at leastone search criteria for use in analyzing the properties of the pluralityof data elements with respect to at least one property of the first dataelement. An analysis module is used to apply the at least one searchcriteria to the properties of the plurality of data elements foridentifying the second data element from the plurality of data elementsand the corresponding at least one association element. The at least oneassociation element is configured for representing a connection betweenthe first data element and the second data element, such that theconnection has a first property common to a property of the first dataelement and a second property common to a property of the second dataelement. A visualization module is used to generate a visualrepresentation of the first and second data elements and the associationelement configured for display on a user interface for subsequentinteraction with user events. The visual representation includes aspatial domain including a reference surface for providing a spatialreference frame having at least two spatial dimensions and a temporaldomain operatively coupled to the spatial domain for providing a commontemporal reference frame for locations of interest in the spatialdomain.

According to a first aspect there is provided a method for analysing aplurality of data elements having both temporal and spatial properties,a first data element and a second data element of the plurality of dataelements linked by at least one association element, the methodcomprising the steps of: selecting the first data element from theplurality of data elements; providing at least one search criteria foruse in analysing the properties of the plurality of data elements withrespect to at least one property of the first data element; applying theat least one search criteria to the properties of the plurality of dataelements for identifying the second data element from the plurality ofdata elements and the corresponding at least one association element,the at least one association element configured for representing aconnection between the first data element and the second data element,the connection having a first property common to a property of the firstdata element and a second property common to a property of the seconddata element; and generating a visual representation of the first andsecond data elements and the association element configured for displayon a user interface for subsequent interaction with user events, thevisual representation including a spatial domain including a referencesurface for providing a spatial reference frame having at least twospatial dimensions and including a temporal domain operatively coupledto the spatial domain for providing a common temporal reference framefor locations of interest in the spatial domain.

According to a second aspect there is provided a system for analysing aplurality of data elements having both temporal and spatial properties,a first data element and a second data element of the plurality of dataelements linked by at least one association element, the systemcomprising: a user interface for selecting the first data element fromthe plurality of data elements and for providing at least one searchcriteria for use in analysing the properties of the plurality of dataelements with respect to at least one property of the first dataelement; an analysis module configured for applying the at least onesearch criteria to the properties of the plurality of data elements foridentifying the second data element from the plurality of data elementsand the corresponding at least one association element, the at least oneassociation element configured for representing a connection between thefirst data element and the second data element, the connection having afirst property common to a property of the first data element and asecond property common to a property of the second data element; and avisualisation module configured for generating a visual representationof the first and second data elements and the association elementconfigured for display on the user interface for subsequent interactionwith user events, the visual representation including a spatial domainincluding a reference surface for providing a spatial reference framehaving at least two spatial dimensions and including a temporal domainoperatively coupled to the spatial domain for providing a commontemporal reference frame for locations of interest in the spatialdomain.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of these and other embodiments of the presentinvention can be obtained with reference to the following drawings anddetailed description of the preferred embodiments, in which:

FIG. 1 is a block diagram of a data processing system for avisualization tool;

FIG. 2 shows further details of the data processing system of FIG. 1;

FIG. 3 shows further details of the visualization tool of FIG. 1;

FIG. 4 shows further details of a visualization representation fordisplay on a visualization interface of the system of FIG. 1;

FIG. 5 is an example visualization representation of FIG. 1 showingEvents in Concurrent Time and Space;

FIG. 6 shows example data objects and associations of FIG. 1;

FIG. 7 shows further example data objects and associations of FIG. 1;

FIG. 8 shows changes in orientation of a reference surface of thevisualization representation of FIG. 1;

FIG. 9 is an example timeline of FIG. 8;

FIG. 10 is a further example timeline of FIG. 8;

FIG. 11 is a further example timeline of FIG. 8 showing a time chart;

FIG. 12 is a further example of the time chart of FIG. 11;

FIG. 13 shows example user controls for the visualization representationof FIG. 5;

FIG. 13 a shows example operation of user controls of FIG. 13;

FIG. 14 shows an example operation of the tool of FIG. 3;

FIG. 15 shows a further example operation of the tool of FIG. 3;

FIG. 16 shows a further example operation of the tool of FIG. 3;

FIG. 17 shows an example visualization representation of FIG. 4containing events and target tracking over space and time showingconnections between events;

FIG. 18 shows an example visualization representation containing eventsand target tracking over space and time showing connections betweenevents on a time chart of FIG. 11, and

FIG. 19 is an example operation of the visualization tool of FIG. 3;

FIG. 20 is a further embodiment of FIG. 18 showing imagery;

FIG. 21 is a further embodiment of FIG. 18 showing imagery in a timechart view;

FIG. 22 shows further detail of the aggregation module of FIG. 3;

FIG. 23 shows an example aggregation result of the module of FIG. 22;

FIG. 24 is a further embodiment of the result of FIG. 23;

FIG. 25 shows a summary chart view of a further embodiment of therepresentation of FIG. 20;

FIG. 26 shows an event comparison for the aggregation module of FIG. 23;

FIG. 27 shows a further embodiment of the tool of FIG. 3;

FIG. 27 a shows a further embodiment of the tool of FIG. 3;

FIG. 28 shows an example operation of the tool of FIG. 27;

FIG. 29 shows a further example of the visualization representation ofFIG. 4;

FIG. 30 is a further example of the charts of FIG. 25;

FIGS. 31 a,b,c,d show example control sliders of analysis functions ofthe tool of FIG. 3;

FIGS. 32 a,b,c,d show example an embodiment of an analysis function ofthe tool of FIG. 27;

FIGS. 33 a,b,c, show further example embodiments of the analysisfunction of the tool of FIG. 27;

FIGS. 34 a,b show further example embodiments of analysis function ofthe tool of FIG. 27; and

FIG. 35 shows a further example embodiment of analysis function of thetool of FIG. 27.

It is noted that similar references are used in different figures todenote similar components.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description of the embodiments of the presentinvention does not limit the implementation of the invention to anyparticular computer programming language. The present invention may beimplemented in any computer programming language provided that the OS(Operating System) provides the facilities that may support therequirements of the present invention. A preferred embodiment isimplemented in the Java computer programming language (or other computerprogramming languages in conjunction with C/C++). Any limitationspresented would be a result of a particular type of operating system,computer programming language, or data processing system and would notbe a limitation of the present invention.

Visualization Environment

Referring to FIG. 1, a visualization data processing system 100 includesa visualization tool 12 for processing a collection of data objects 14as input data elements to a user interface 202. The data objects 14 arecombined with a respective set of associations 16 by the tool 12 togenerate an interactive visual representation 18 on the visual interface(VI) 202. The data objects 14 include event objects 20, location objects22, images 23 and entity objects 24, as further described below. The setof associations 16 include individual associations 26 that associatetogether various subsets of the objects 20, 22, 23, 24, as furtherdescribed below. Management of the data objects 14 and set ofassociations 16 are driven by user events 109 of a user (not shown) viathe user interface 108 (see FIG. 2) during interaction with the visualrepresentation 18. The representation 18 shows connectivity betweentemporal and spatial information of data objects 14 at multi-locationswithin the spatial domain 400 (see FIG. 4).

Data Processing System

Referring to FIG. 2, the data processing system 100 has a user interface108 for interacting with the tool 12, the user interface 108 beingconnected to a memory 102 via a BUS 106. The interface 108 is coupled toa processor 104 via the BUS 106, to interact with user events 109 tomonitor or otherwise instruct the operation of the tool 12 via anoperating system 110. The user interface 108 can include one or moreuser input devices such as but not limited to a QWERTY keyboard, akeypad, a trackwheel, a stylus, a mouse, and a microphone. The visualinterface 202 is considered the user output device, such as but notlimited to a computer screen display. If the screen is touch sensitive,then the display can also be used as the user input device as controlledby the processor 104. Further, it is recognized that the data processingsystem 100 can include a computer readable storage medium 46 coupled tothe processor 104 for providing instructions to the processor 104 and/orthe tool 12. The computer readable medium 46 can include hardware and/orsoftware such as, by way of example only, magnetic disks, magnetic tape,optically readable medium such as CD/DVD ROMS, and memory cards. In eachcase, the computer readable medium 46 may take the form of a small disk,floppy diskette, cassette, hard disk drive, solid-state memory card, orRAM provided in the memory 102. It should be noted that the above listedexample computer readable mediums 46 can be used either alone or incombination.

Referring again to FIG. 2, the tool 12 interacts via link 116 with a VImanager 112 (also known as a visualization renderer) of the system 100for presenting the visual representation 18 on the visual interface 202.The tool 12 also interacts via link 118 with a data manager 114 of thesystem 100 to coordinate management of the data objects 14 andassociation set 16 from data files or tables 122 of the memory 102. Itis recognized that the objects 14 and association set 16 could be storedin the same or separate tables 122, as desired. The data manager 114 canreceive requests for storing, retrieving, amending, or creating theobjects 14 and association set 16 via the tool 12 and/or directly vialink 120 from the VI manager 112, as driven by the user events 109and/or independent operation of the tool 12. The data manager 114manages the objects 14 and association set 16 via link 123 with thetables 122. Accordingly, the tool 12 and managers 112, 114 coordinatethe processing of data objects 14, association set 16 and user events109 with respect to the content of the screen representation 18displayed in the visual interface 202.

Referring to FIGS. 1, 27, 27 a, and 29, the tool 12 has an informationmodule 712 for generating object information 714 a,b,c,d for display bythe visualization manager 300, in response to user manipulations via theI/O interface 108. The information module 712 has an identificationmodule 750 for recognizing the object type (e.g. entity, event,location, association, etc.) of the selected object(s) 14 from thevisualization representation 18. Once the object 14 type is determined,the identification module 750 uses a rule set 752 to determine thespecified type and display format (e.g. text size/font/colour, images,icons, graphic objects, bounding box size and position on representation18, etc.) 756 of object information 714 a,b,c,d to be displayed on thevisualization representation 18 with respect to the selected object(s)14. The selected object(s) 14 and their specified type and displayformat of the object information 714 a,b,c,d is supplied to an accessmodule 753, which then accesses the tables 122 of the memory 102 forretrieving the object data 754 representing the desired objectinformation 714 a,b,c,d. The access module 753 can also be configured tosupply the retrieved object data 754 and its corresponding displayformat 756 to the visualization manager 300 for effecting display on thevisualization representation 18, shown by example in FIG. 29.

For example, when a mouse pointer 713 (or other user implemented triggerevent—e.g. keyboard identification of selected object(s) 14) is heldover the visual element 410,412 of the representation 18, somepredefined information 714 a,b,c,d is displayed about that selectedvisual element 410,412. The information module 712 is configured todisplay the type of information dependent upon whether the object is aplace 22, target 24, elementary or compound event 20, for example. Forexample, when the place 22 type is selected, the displayed information714 a is formatted by the information module 712 to include such as butnot limited to; Label (e.g. Rome), Attributes attached to the object (ifany); and events associated with that place 22. For example, when thetarget 24/target trail 412 (see FIG. 17) type is selected, the displayedinformation 714 b is formatted by the information module 712 to includesuch as but not limited to; Label, Attributes (if any), eventsassociated with that target 24, as well as the target's icon (if one isassociated with the target 24) is shown. For example, when an elementaryevent 20 a type is selected, the displayed information 714 c isformatted by the information module 712 to include such as but notlimited to; Label, Class, Date, Type, Comment (including Attributes, ifany), associated Targets 24 and Place 22. For example, when a compoundevent 20 b type is selected, the displayed information 714 d isformatted by the information module 712 to include such as but notlimited to; Label, Class, Date, Type, Comment (including Attributes, ifany) and all elementary event popup data for each child event.Accordingly, it is recognized that the information module 712 isconfigured to select data for display from the database tables 122 (seeFIG. 2) appropriate to the type of visual element 410,412 selected bythe user from the visual representation 18.

Tool Information Model

Referring to FIG. 1, a tool information model is composed of the fourbasic data elements (objects 20, 22, 23, 24 and associations 26) thatcan have corresponding display elements in the visual representation 18.The four elements are used by the tool 12 to describe interconnectedactivities and information in time and space as the integrated visualrepresentation 18, as further described below.

Event Data Objects 20

Events are data objects 20 that represent any action that can bedescribed. The following are examples of events;

Bill was at Toms house at 3 pm,

Tom phoned Bill on Thursday,

A tree fell in the forest at 4:13 am, Jun. 3, 1993 and

Tom will move to Spain in the summer of 2004.

The Event is related to a location and a time at which the action tookplace, as well as several data properties and display propertiesincluding such as but not limited to; a short text label, description,location, start-time, end-time, general event type, icon reference,visual layer settings, priority, status, user comment, certainty value,source of information, and default+user-set color. The event data object20 can also reference files such as images or word documents.

Locations and times may be described with varying precision. Forexample, event times can be described as “during the week of January4^(th)” or “in the month of September”. Locations can be described as“Spain” or as “New York” or as a specific latitude and longitude.

Entity Data Objects 24

Entities are data objects 24 that represent any thing related to orinvolved in an event, including such as but not limited to; people,objects, organizations, equipment, businesses, observers, affiliationsetc. Data included as part of the Entity data object 24 can be shorttext label, description, general entity type, icon reference, visuallayer settings, priority, status, user comment, certainty value, sourceof information, and default+user-set color. The entity data can alsoreference files such as images or word documents. It is recognized inreference to FIGS. 6 and 7 that the term Entities includes “People”, aswell as equipment (e.g. vehicles), an entire organization (e.g.corporate entity), currency, and any other object that can be trackedfor movement in the spatial domain 400. It is also recognized that theentities 24 could be stationary objects such as but not limited tobuildings. Further, entities can be phone numbers and web sites. To beexplicit, the entities 24 as given above by example only can be regardedas Actors

Location Data Objects 22

Locations are data objects 22 that represent a place within a spatialcontext/domain, such as a geospatial map, a node in a diagram such as aflowchart, or even a conceptual place such as “Shang-ri-la” or other“locations” that cannot be placed at a specific physical location on amap or other spatial domain. Each Location data object 22 can store suchas but not limited to; position coordinates, a label, description, colorinformation, precision information, location type, non-geospatial flagand user comments.

Associations

Event 20, Location 22 and Entity 24 are combined into groups or subsetsof the data objects 14 in the memory 102 (see FIG. 2) using associations26 to describe real-world occurrences. The association is defined as aninformation object that describes a pairing between 2 data objects 14.For example, in order to show that a particular entity was present whenan event occurred, the corresponding association 26 is created torepresent that Entity X “was present at” Event A. For example,associations 26 can include such as but not limited to; describing acommunication connection between two entities 24, describing a physicalmovement connection between two locations of an entity 24, and arelationship connection between a pair of entities 24 (e.g. familyrelated and/or organizational related). It is recognised that theassociations 26 can describe direct and indirect connections. Otherexamples can include phone numbers and web sites.

A variation of the association type 26 can be used to define a subclassof the groups 27 to represent user hypotheses. In other words, groups 27can be created to represent a guess or hypothesis that an eventoccurred, that it occurred at a certain location or involved certainentities. Currently, the degree of belief/accuracy/evidence reliabilitycan be modeled on a simple 1-2-3 scale and represented graphically withline quality on the visual representation 18.

Image Data Objects 23

Standard icons for data objects 14 as well as small images 23 for suchas but not limited to objects 20,22,24 can be used to describe entitiessuch as people, organizations and objects. Icons are also used todescribe activities. These can be standard or tailored icons, or actualimages of people, places, and/or actual objects (e.g. buildings).Imagery can be used as part of the event description. Images 23 can beviewed in all of the visual representation 18 contexts, as for exampleshown in FIGS. 20 and 21 which show the use of images 23 in the timelines 422 and the time chart 430 views. Sequences of images 23 can beanimated to help the user detect changes in the image over time andspace.

Annotations 21

Annotations 21 in Geography and Time (see FIG. 22) can be represented asmanually placed lines or other shapes (e.g. pen/pencil strokes) can beplaced on the visual representation 18 by an operator of the tool 12 andused to annotate elements of interest with such as but not limited toarrows, circles and freeform markings. Some examples are shown in FIG.21. These annotations 21 are located in geography (e.g. spatial domain400) and time (e.g. temporal domain 422) and so can appear and disappearon the visual representation 18 as geographic and time contexts arenavigated through the user input events 109.

Visualization Tool 12

Referring to FIG. 3, the visualization tool 12 has a visualizationmanager 300 for interacting with the data objects 14 for presentation tothe interface 202 via the VI manager 112. The Data Objects 14 are formedinto groups 27 through the associations 26 and processed by theVisualization Manager 300. The groups 27 comprise selected subsets ofthe objects 20, 21, 22, 23, 24 combined via selected associations 26.This combination of data objects 14 and association sets 16 can beaccomplished through predefined groups 27 added to the tables 122 and/orthrough the user events 109 during interaction of the user directly withselected data objects 14 and association sets 16 via the controls 306.It is recognized that the predefined groups 27 could be loaded into thememory 102 (and tables 122) via the computer readable medium 46 (seeFIG. 2). The Visualization manager 300 also processes user event 109input through interaction with a time slider and other controls 306,including several interactive controls for supporting navigation andanalysis of information within the visual representation 18 (see FIG. 1)such as but not limited to data interactions of selection, filtering,hide/show and grouping as further described below. Use of the groups 27is such that subsets of the objects 14 can be selected and groupedthrough associations 26. In this way, the user of the tool 12 canorganize observations into related stories or story fragments. Thesegroupings 27 can be named with a label and visibility controls, whichprovide for selected display of the groups 27 on the representation 18,e.g. the groups 27 can be turned on and off with respect to display tothe user of the tool 12.

The Visualization Manager 300 processes the translation from raw dataobjects 14 to the visual representation 18. First, Data Objects 14 andassociations 16 can be formed by the Visualization Manager 300 into thegroups 27, as noted in the tables 122, and then processed. TheVisualization Manager 300 matches the raw data objects 14 andassociations 16 with sprites 308 (i.e. visual processingobjects/components that know how to draw and render visual elements forspecified data objects 14 and associations 16) and sets a drawingsequence for implementation by the VI manager 112. The sprites 308 arevisualization components that take predetermined information schema asinput and output graphical elements such as lines, text, images andicons to the computers graphics system. Entity 24, event 20 and location22 data objects each can have a specialized sprite 308 type designed torepresent them. A new sprite instance is created for each entity, eventand location instance to manage their representation in the visualrepresentation 18 on the display.

The sprites 308 are processed in order by the visualization manager 300,starting with the spatial domain (terrain) context and locations,followed by Events and Timelines, and finally Entities. Timelines aregenerated and Events positioned along them. Entities are rendered lastby the sprites 308 since the entities depend on Event positions. It isrecognised that processing order of the sprites 308 can be other than asdescribed above.

The Visualization manager 112 renders the sprites 308 to create thefinal image including visual elements representing the data objects 14and associates 16 of the groups 27, for display as the visualrepresentation 18 on the interface 202. After the visual representation18 is on the interface 202, the user event 109 inputs flow into theVisualization Manager, through the VI manager 112 and cause the visualrepresentation 18 to be updated. The Visualization Manager 300 can beoptimized to update only those sprites 308 that have changed in order tomaximize interactive performance between the user and the interface 202.

Layout of the Visualization Representation 18

The visualization technique of the visualization tool 12 is designed toimprove perception of entity activities, movements and relationships asthey change over time in a concurrent time-geographic ortime-diagrammatical context. The visual representation 18 of the dataobjects 14 and associations 16 consists of a combined temporal-spatialdisplay to show interconnecting streams of events over a range of timeon a map or other schematic diagram space, both hereafter referred to incommon as a spatial domain 400 (see FIG. 4). Events can be representedwithin an X,Y,T coordinate space, in which the X,Y plane shows thespatial domain 400 (e.g. geographic space) and the Z-axis represents atime series into the future and past, referred to as a temporal domain402. In addition to providing the spatial context, a reference surface(or reference spatial domain) 404 marks an instant of focus betweenbefore and after, such that events “occur” when they meet the surface ofthe ground reference surface 404. FIG. 4 shows how the visualizationmanager 300 (see FIG. 3) combines individual frames 406 (spatial domains400 taken at different times Ti 407) of event/entity/location visualelements 410, which are translated into a continuous integrated spatialand temporal visual representation 18. It should be noted connectionvisual elements 412 can represent presumed location (interpolated) ofEntity between the discrete event/entity/location represented by thevisual elements 410. Another interpretation for connections elements 412could be signifying communications between different Entities atdifferent locations, which are related to the same event as furtherdescribed below.

Referring to FIG. 5, an example visual representation 18 visuallydepicts events over time and space in an x, y, t space (or x, y, z, tspace with elevation data). The example visual representation 18generated by the tool 12 (see FIG. 2) is shown having the time domain402 as days in April, and the spatial domain 400 as a geographical mapproviding the instant of focus (of the reference surface 404) assometime around noon on April 23—the intersection point between thetimelines 422 and the reference surface 404 represents the instant offocus. The visualization representation 18 represents the temporal 402,spatial 400 and connectivity elements 412 (between two visual elements410) of information within a single integrated picture on the interface202 (see FIG. 1). Further, the tool 12 provides an interactive analysistool for the user with interface controls 306 to navigate the temporal,spatial and connectivity dimensions. The tool 12 is suited to theinterpretation of any information in which time, location andconnectivity are key dimensions that are interpreted together. Thevisual representation 18 is used as a visualization technique fordisplaying and tracking events, people, and equipment within thecombined temporal and spatial domains 402, 400 display. Tracking andanalyzing entities 24 and streams has traditionally been the domain ofinvestigators, whether that be police services or military intelligence.In addition, business users also analyze events 20 in time and spatialdomains 400, 402 to better understand phenomenon such as customerbehavior or transportation patterns. The visualization tool 12 can beapplied for both reporting and analysis.

The visual representation 18 can be applied as an analyst workspace forexploration, deep analysis and presentation for such as but not limitedto:

-   -   Situations involving people and organizations that interact over        time and in which geography or territory plays a role;    -   Storing and reviewing activity reports over a given period. Used        in this way the representation 18 could provide a means to        determine a living history, context and lessons learned from        past events; and    -   As an analysis and presentation tool for long term tracking and        surveillance of persons and equipment activities.

The visualization tool 12 provides the visualization representation 18as an interactive display, such that the users (e.g. intelligenceanalysts, business marketing analysts) can view, and work with, largenumbers of events. Further, perceived patterns, anomalies andconnections can be explored and subsets of events can be grouped into“story” or hypothesis fragments. The visualization tool 12 includes avariety of capabilities such as but not limited to:

-   -   An event-based information architecture with places, events,        entities (e.g. people) and relationships;    -   Past and future time visibility and animation controls;    -   Data input wizards for describing single events and for loading        many events from a table;    -   Entity and event connectivity analysis in time and geography;    -   Path displays in time and geography;    -   Configurable workspaces allowing ad hoc, drag and drop        arrangements of events;    -   Search, filter and drill down tools;    -   Creation of sub-groups and overlays by selecting events and        dragging them into sets (along with associated spatial/time        scope properties); and    -   Adaptable display functions including dynamic show/hide        controls.        Example Objects 14 with Associations 16

In the visualization tool 12, specific combinations of associated dataelements (objects 20, 22, 24 and associations 26) can be defined. Thesedefined groups 27 are represented visually as visual elements 410 inspecific ways to express various types of occurrences in the visualrepresentation 18. The following are examples of how the groups 27 ofassociated data elements can be formed to express specific occurrencesand relationships shown as the connection visual elements 412.

Referring to FIGS. 6 and 7, example groups 27 (denoting common realworld occurrences) are shown with selected subsets of the objects 20,22, 24 combined via selected associations 26. The correspondingvisualization representation 18 is shown as well including the temporaldomain 402, the spatial domain 400, connection visual elements 412 andthe visual elements 410 representing the event/entity/locationcombinations. It is noted that example applications of the groups 27 aresuch as but not limited to those shown in FIGS. 6 and 7. In the FIGS. 6and 7 it is noted that event objects 20 are labeled as “Event 1”, “Event2”, location objects 22 are labeled as “Location A”, “Location B”, andentity objects 24 are labeled as “Entity X”, “Entity Y”. The set ofassociations 16 are labeled as individual associations 26 withconnections labeled as either solid or dotted lines 412 between twoevents, or dotted in the case of an indirect connection between twolocations.

Visual Elements Corresponding to Spatial and Temporal Domains

The visual elements 410 and 412, their variations and behaviorfacilitate interpretation of the concurrent display of events in thetime 402 and space 400 domains. In general, events reference thelocation at which they occur and a list of Entities and their role inthe event. The time at which the event occurred or the time span overwhich the event occurred are stored as parameters of the event.

Spatial Domain Representation

Referring to FIG. 8, the primary organizing element of the visualizationrepresentation 18 is the 2D/3D spatial reference frame (subsequentlyincluded herein with reference to the spatial domain 400). The spatialdomain 400 consists of a true 2D/3D graphics reference surface 404 inwhich a 2D or 3 dimensional representation of an area is shown. Thisspatial domain 400 can be manipulated using a pointer device (notshown—part of the controls 306—see FIG. 3) by the user of the interface108 (see FIG. 2) to rotate the reference surface 404 with respect to aviewpoint 420 or viewing ray extending from a viewer 423. The user (i.e.viewer 423) can also navigate the reference surface 404 by scrolling inany direction, zooming in or out of an area and selecting specific areasof focus. In this way the user can specify the spatial dimensions of anarea of interest the reference surface 404 in which to view events intime. The spatial domain 400 represents space essentially as a plane(e.g. reference surface 404), however is capable of representing 3dimensional relief within that plane in order to express geographicalfeatures involving elevation. The spatial domain 400 can be madetransparent so that timelines 422 of the temporal domain 402 can extendbehind the reference surface 404 are still visible to the user. FIG. 8shows how the viewer 423 facing timelines 422 can rotate to face theviewpoint 420 no matter how the reference surface 404 is rotated in 3dimensions with respect to the viewpoint 420.

The spatial domain 400 includes visual elements 410, 412 (see FIG. 4)that can represent such as but not limited to map information, digitalelevation data, diagrams, and images used as the spatial context. Thesetypes of spaces can also be combined into a workspace. The user can alsocreate diagrams using drawing tools (of the controls 306—see FIG. 3)provided by the visualization tool 12 to create custom diagrams andannotations within the spatial domain 400.

Event Representation and Interactions

Referring to FIGS. 4 and 8, events are represented by a glyph, or iconas the visual element 410, placed along the timeline 422 at the point intime that the event occurred. The glyph can be actually a group ofgraphical objects, or layers, each of which expresses the content of theevent data object 20 (see FIG. 1) in a different way. Each layer can betoggled and adjusted by the user on a per event basis, in groups oracross all event instances. The graphical objects or layers for eventvisual elements 410 are such as but not limited to:

1. Text Label

-   -   The Text label is a text graphic meant to contain a short        description of the event content. This text always faces the        viewer 423 no matter how the reference surface 404 is oriented.        The text label incorporates a de-cluttering function that        separates it from other labels if they overlap. When two events        are connected with a line (see connections 412 below) the label        will be positioned at the midpoint of the connection line        between the events. The label will be positioned at the end of a        connection line that is clipped at the edge of the display area.

2. Indicator—Cylinder, Cube or Sphere

-   -   The indicator marks the position in time. The color of the        indicator can be manually set by the user in an event properties        dialog. Color of event can also be set to match the Entity that        is associated with it. The shape of the event can be changed to        represent different aspect of information and can be set by the        user. Typically it is used to represent a dimension such as type        of event or level of importance.

3. Icon

-   -   An icon or image can also be displayed at the event location.        This icon/image 23 may used to describe some aspect of the        content of the event. This icon/image 23 may be user-specified        or entered as part of a data file of the tables 122 (see FIG.        2).

4. Connection Elements 412

-   -   Connection elements 412 can be lines, or other geometrical        curves, which are solid or dashed lines that show connections        from an event to another event, place or target. A connection        element 412 may have a pointer or arrowhead at one end to        indicate a direction of movement, polarity, sequence or other        vector-like property. If the connected object is outside of the        display area, the connection element 412 can be coupled at the        edge of the reference surface 404 and the event label will be        positioned at the clipped end of the connection element 412.

5. Time Range Indicator

-   -   A Time Range Indicator (not shown) appears if an event occurs        over a range of time. The time range can be shown as a line        parallel to the timeline 422 with ticks at the end points. The        event Indicator (see above) preferably always appears at the        start time of the event.

The Event visual element 410 can also be sensitive to interaction. Thefollowing user events 109 via the user interface 108 (see FIG. 2) arepossible, such as but not limited to:

Mouse-Left-Click:

Selects the visual element 410 of the visualization representation 18 onthe VI 202 (see FIG. 2) and highlights it, as well as simultaneouslydeselecting any previously selected visual element 410, as desired.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

-   -   Adds the visual element 410 to an existing selection set.

Mouse-Left-Double-Click:

-   -   Opens a file specified in an event data parameter if it exists.        The file will be opened in a system-specified default        application window on the interface 202 based on its file type.

Mouse-Right-Click:

-   -   Displays an in-context popup menu with options to hide, delete        and set properties.

Mouse Over Drilldown:

-   -   When the mouse pointer (not shown) is placed over the indicator,        a text window is displayed next to the pointer, showing        information about the visual element 410. When the mouse pointer        is moved away from the indicator, the text window disappears.

Location Representation

Locations are visual elements 410 represented by a glyph, or icon,placed on the reference surface 404 at the position specified by thecoordinates in the corresponding location data object 22 (see FIG. 1).The glyph can be a group of graphical objects, or layers, each of whichexpresses the content of the location data object 22 in a different way.Each layer can be toggled and adjusted by the user on a per Locationbasis, in groups or across all instances. The visual elements 410 (e.g.graphical objects or layers) for Locations are such as but not limitedto:

1. Text Label

-   -   The Text label is a graphic object for displaying the name of        the location. This text always faces the viewer 422 no matter        how the reference surface 404 is oriented. The text label        incorporates a de-cluttering function that separates it from        other labels if they overlap.

2. Indicator

-   -   The indicator is an outlined shape that marks the position or        approximate position of the Location data object 22 on the        reference surface 404. There are, such as but not limited to, 7        shapes that can be selected for the locations visual elements        410 (marker) and the shape can be filled or empty. The outline        thickness can also be adjusted. The default setting can be a        circle and can indicate spatial precision with size. For        example, more precise locations, such as addresses, are smaller        and have thicker line width, whereas a less precise location is        larger in diameter, but uses a thin line width.    -   The Location visual elements 410 are also sensitive to        interaction. The following interactions are possible:

Mouse-Left-Click:

-   -   Selects the location visual element 410 and highlights it, while        deselecting any previously selected location visual elements        410.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

-   -   Adds the location visual element 410 to an existing selection        set.

Mouse-Left-Double-Click:

-   -   Opens a file specified in a Location data parameter if it        exists. The file will be opened in a system-specified default        application window based on its file type.

Mouse-Right-Click:

-   -   Displays an in-context popup menu with options to hide, delete        and set properties of the location visual element 410.

Mouse Over Drilldown:

-   -   When the Mouse pointer is placed over the location indicator, a        text window showing information about the location visual        element 410 is displayed next to the pointer. When the mouse        pointer is moved away from the indicator, the text window        disappears.

Mouse-Left-Click-Hold-and-Drag:

-   -   Interactively repositions the location visual element 410 by        dragging it across the reference surface 404.

Non-Spatial Locations

Locations 22 have the ability to represent indeterminate position. Theseare referred to as non-spatial locations 22. Locations 22 tagged asnon-spatial can be displayed at the edge of the reference surface 404just outside of the spatial context of the spatial domain 400. Thesenon-spatial or virtual locations 22 can be always visible no matterwhere the user is currently zoomed in on the reference surface 404.Events and Timelines 422 that are associated with non-spatial Locations22 can be rendered the same way as Events with spatial Locations 22.

Further, it is recognized that spatial locations 22 can representactual, physical places, such that if the latitude/longitude is knownthe location 22 appears at that position on the map or if thelatitude/longitude is unknown the location 22 appears on the bottomcorner of the map (for example). Further, it is recognized thatnon-spatial locations 22 can represent places with no real physicallocation and can always appear off the right side of map (for example).For events 20, if the location 22 of the event 20 is known, the location22 appears at that position on the map. However, if the location 22 isunknown, the location 22 can appear halfway (for example) between thegeographical positions of the adjacent event locations 22 (e.g. part oftarget tracking).

Entity Representation

Entity visual elements 410 are represented by a glyph, or icon, and canbe positioned on the reference surface 404 or other area of the spatialdomain 400, based on associated Event data that specifies its positionat the current Moment of Interest 900 (see FIG. 9) (i.e. specific pointon the timeline 422 that intersects the reference surface 404). If thecurrent Moment of Interest 900 lies between 2 events in time thatspecify different positions, the Entity position will be interpolatedbetween the 2 positions. Alternatively, the Entity could be positionedat the most recent known location on he reference surface 404. TheEntity glyph is actually a group of the entity visual elements 410 (e.g.graphical objects, or layers) each of which expresses the content of theevent data object 20 in a different way. Each layer can be toggled andadjusted by the user on a per event basis, in groups or across all eventinstances. The entity visual elements 410 are such as but not limitedto:

1. Text Label

-   -   The Text label is a graphic object for displaying the name of        the Entity. This text always faces the viewer no matter how the        reference surface 404 is oriented. The text label incorporates a        de-cluttering function that separates it from other labels if        they overlap.

2. Indicator

-   -   The indicator is a point showing the interpolated or real        position of the Entity in the spatial context of the reference        surface 404. The indicator assumes the color specified as an        Entity color in the Entity data model.

3. Image Icon

-   -   An icon or image is displayed at the Entity location. This icon        may used to represent the identity of the Entity. The displayed        image can be user-specified or entered as part of a data file.        The Image Icon can have an outline border that assumes the color        specified as the Entity color in the Entity data model. The        Image Icon incorporates a de-cluttering function that separates        it from other Entity Image Icons if they overlap.

4. Past Trail

-   -   The Past Trail is the connection visual element 412, as a series        of connected lines that trace previous known positions of the        Entity over time, starting from the current Moment of Interest        900 and working backwards into past time of the timeline 422.        Previous positions are defined as Events where the Entity was        known to be located. The Past Trail can mark the path of the        Entity over time and space simultaneously.

5. Future Trail

-   -   The Future Trail is the connection visual element 412, as a        series of connected lines that trace future known positions of        the Entity over time, starting from the current Moment of        Interest 900 and working forwards into future time. Future        positions are defined as Events where the Entity is known to be        located. The Future Trail can mark the future path of the Entity        over time and space simultaneously.

The Entity representation is also sensitive to interaction. Thefollowing interactions are possible, such as but not limited to:

Mouse-Left-Click:

-   -   Selects the entity visual element 410 and highlights it and        deselects any previously selected entity visual element 410.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

-   -   Adds the entity visual element 410 to an existing selection set

Mouse-Left-Double-Click:

-   -   Opens the file specified in an Entity data parameter if it        exists. The file will be opened in a system-specified default        application window based on its file type.

Mouse-Right-Click:

-   -   Displays an in-context popup menu with options to hide, delete        and set properties of the entity visual element 410.

Mouse Over Drilldown:

-   -   When the Mouse pointer is placed over the indicator, a text        window showing information about the entity visual element 410        is displayed next to the pointer. When the mouse pointer is        moved away from the indicator, the text window disappears.        Temporal Domain including Timelines

Referring to FIGS. 8 and 9, the temporal domain provides a commontemporal reference frame for the spatial domain 400, whereby the domains400, 402 are operatively coupled to one another to simultaneouslyreflect changes in interconnected spatial and temporal properties of thedata elements 14 and associations 16. Timelines 422 (otherwise known astime tracks) represent a distribution of the temporal domain 402 overthe spatial domain 400, and are a primary organizing element ofinformation in the visualization representation 18 that make it possibleto display events across time within the single spatial display on theVI 202 (see FIG. 1). Timelines 422 represent a stream of time through aparticular Location visual element 410 a positioned on the referencesurface 404 and can be represented as a literal line in space. Otheroptions for representing the timelines/time tracks 422 are such as butnot limited to curved geometrical shapes (e.g. spirals) including 2D and3D curves when combining two or more parameters in conjuction with thetemporal dimension. Each unique Location of interest (represented by thelocation visual element 410 a) has one Timeline 422 that passes throughit. Events (represented by event visual elements 410 b) that occur atthat Location are arranged along this timeline 422 according to theexact time or range of time at which the event occurred. In this waymultiple events (represented by respective event visual elements 410 b)can be arranged along the timeline 422 and the sequence made visuallyapparent. A single spatial view will have as many timelines 422 asnecessary to show every Event at every location within the currentspatial and temporal scope, as defined in the spatial 400 and temporal402 domains (see FIG. 4) selected by the user. In order to makecomparisons between events and sequences of event between locations, thetime range represented by multiple timelines 422 projecting through thereference surface 404 at different spatial locations is synchronized. Inother words the time scale is the same across all timelines 422 in thetime domain 402 of the visual representation 18. Therefore, it isrecognised that the timelines 422 are used in the visual representation18 to visually depict a graphical visualization of the data objects 14over time with respect to their spatial properties/attributes.

For example, in order to make comparisons between events 20 andsequences of events 20 between locations 410 of interest (see FIG. 4),the time range represented by the timelines 422 can be synchronized. Inother words, the time scale can be selected as the same for everytimeline 422 of the selected time range of the temporal domain 402 ofthe representation 18.

Representing Current, Past and Future

Three distinct strata of time are displayed by the timelines 422,namely;

-   1. The “moment of interest” 900 or browse time, as selected by the    user,-   2. a range 902 of past time preceding the browse time called “past”,    and-   3. a range 904 of time after the moment of interest 900, called    “future”

On a 3D Timeline 422, the moment of focus 900 is the point at which thetimeline intersects the reference surface 404. An event that occurs atthe moment of focus 900 will appear to be placed on the referencesurface 404 (event representation is described above). Past and futuretime ranges 902, 904 extend on either side (above or below) of themoment of interest 900 along the timeline 422. Amount of time into thepast or future is proportional to the distance from the moment of focus900. The scale of time may be linear or logarithmic in either direction.The user may select to have the direction of future to be down and pastto be up or vice versa.

There are three basic variations of Spatial Timelines 422 that emphasizespatial and temporal qualities to varying extents. Each variation has aspecific orientation and implementation in terms of its visualconstruction and behavior in the visualization representation 18 (seeFIG. 1). The user may choose to enable any of the variations at any timeduring application runtime, as further described below.

3D Z-Axis Timelines

FIG. 10 shows how 3D Timelines 422 pass through reference surface 404locations 410 a. 3D timelines 422 are locked in orientation (angle) withrespect to the orientation of the reference surface 404 and are affectedby changes in perspective of the reference surface 404 about theviewpoint 420 (see FIG. 8). For example, the 3D Timelines 422 can beoriented normal to the reference surface 404 and exist within itscoordinate space. Within the 3D spatial domain 400, the referencesurface 404 is rendered in the X-Y plane and the timelines 422 runparallel to the Z-axis through locations 410 a on the reference surface404. Accordingly, the 3D Timelines 422 move with the reference surface404 as it changes in response to user navigation commands and viewpointchanges about the viewpoint 420, much like flag posts are attached tothe ground in real life. The 3D timelines 422 are subject to the sameperspective effects as other objects in the 3D graphical window of theVI 202 (see FIG. 1) displaying the visual representation 18. The 3DTimelines 422 can be rendered as thin cylindrical volumes and arerendered only between events 410 a with which it shares a location andthe location 410 a on the reference surface 404. The timeline 422 mayextend above the reference surface 404, below the reference surface 404,or both. If no events 410 b for its location 410 a are in view thetimeline 422 is not shown on the visualization representation 18.

3D Viewer Facing Timelines

Referring to FIG. 8, 3D Viewer-facing Timelines 422 are similar to 3DTimelines 422 except that they rotate about a moment of focus 425 (pointat which the viewing ray of the viewpoint 420 intersects the referencesurface 404) so that the 3D Viewer-facing Timeline 422 always remainperpendicular to viewer 423 from which the scene is rendered. 3DViewer-facing Timelines 422 are similar to 3D Timelines 422 except thatthey rotate about the moment of focus 425 so that they are alwaysparallel to a plane 424 normal to the viewing ray between the viewer 423and the moment of focus 425. The effect achieved is that the timelines422 are always rendered to face the viewer 423, so that the length ofthe timeline 422 is always maximized and consistent. This techniqueallows the temporal dimension of the temporal domain 402 to be read bythe viewer 423 indifferent to how the reference surface 404 many beoriented to the viewer 423. This technique is also generally referred toas “billboarding” because the information is always oriented towards theviewer 423. Using this technique the reference surface 404 can be viewedfrom any direction (including directly above) and the temporalinformation of the timeline 422 remains readable.

Linked TimeChart Timelines

Referring to FIG. 11, showing how an overlay time chart 430 is connectedto the reference surface 404 locations 410 a by timelines 422. Thetimelines 422 of the Linked TimeChart 430 are timelines 422 that connectthe 2D chart 430 (e.g. grid) in the temporal domain 402 to locations 410a marked in the 3D spatial domain 400. The timeline grid 430 is renderedin the visual representation 18 as an overlay in front of the 2D or 3Dreference surface 404. The timeline chart 430 can be a rectangularregion containing a regular or logarithmic time scale upon which eventrepresentations 410 b are laid out. The chart 430 is arranged so thatone dimension 432 is time and the other is location 434 based on theposition of the locations 410 a on the reference surface 404. As thereference surface 404 is navigated or manipulated the timelines 422 inthe chart 430 move to follow the new relative location 410 a positions.This linked location and temporal scrolling has the advantage that it iseasy to make temporal comparisons between events since time isrepresented in a flat chart 430 space. The position 410 b of the eventcan always be traced by following the timeline 422 down to the referencesurface 404 to the location 410 a.

Referring to FIGS. 11 and 12, the TimeChart 430 can be rendered in 2orientations, one vertical and one horizontal. In the vertical mode ofFIG. 11, the TimeChart 430 has the location dimension 434 shownhorizontally, the time dimension 432 vertically, and the timelines 422connect vertically to the reference surface 404. In the horizontal modeof FIG. 12, the TimeChart 430 has the location dimension 434 shownvertically, the time dimension 432 shown horizontally and the timelines422 connect to the reference surface 404 horizontally. In both cases theTimeChart 430 position in the visualization representation 18 can bemoved anywhere on the screen of the VI 202 (see FIG. 1), so that thechart 430 may be on either side of the reference surface 404 or in frontof the reference surface 404. In addition, the temporal directions ofpast 902 and future 904 can be swapped on either side of the focus 900.

Interaction Interface Descriptions

Referring to FIGS. 3 and 13, several interactive controls 306 supportnavigation and analysis of information within the visualizationrepresentation 12, as monitored by the visualization manger 300 inconnection with user events 109. Examples of the controls 306 are suchas but not limited to a time slider 910, an instant of focus selector912, a past time range selector 914, and a future time selector 916. Itis recognized that these controls 306 can be represented on the VI 202(see FIG. 1) as visual based controls, text controls, and/or acombination thereof.

Time and Range Slider 910

Referring to FIG. 13, the timeline slider 910 is a linear time scalethat is visible underneath the visualization representation 18(including the temporal 402 and spatial 400 domains). The control 910contains sub controls/selectors that allow control of three independenttemporal parameters: the Instant of Focus, the Past Range of Time andthe Future Range of Time.

Continuous animation of events 20 over time and geography can beprovided as the time slider 910 is moved forward and backwards in time.Example, if a vehicle moves from location

A at t1 to location B at t2, the vehicle (object 23,24) is shown movingcontinuously across the spatial domain 400 (e.g. map). The timelines 422can animate up and down at a selected frame rate in association withmovement of the slider 910.

Instant of Focus

The instant of focus selector 912 is the primary temporal control. It isadjusted by dragging it left or right with the mouse pointer across thetime slider 910 to the desired position. As it is dragged, the Past andFuture ranges move with it. The instant of focus 900 (see FIG. 12) (alsoknown as the browse time) is the moment in time represented at thereference surface 404 in the spatial-temporal visualizationrepresentation 18. As the instant of focus selector 912 is moved by theuser forward or back in time along the slider 910, the visualizationrepresentation 18 displayed on the interface 202 (see FIG. 1) updatesthe various associated visual elements of the temporal 402 and spatial400 domains to reflect the new time settings. For example, placement ofEvent visual elements 410 animate along the timelines 422 and Entityvisual elements 410 move along the reference surface 404 interpolatingbetween known locations visual elements 410 (see FIGS. 6 and 7).Examples of movement are given with reference to FIGS. 14, 15, and 16below.

Past Time Range

The Past Time Range selector 914 sets the range of time before themoment of interest 900 (see FIG. 11) for which events will be shown. ThePast Time range is adjusted by dragging the selector 914 left and rightwith the mouse pointer. The range between the moment of interest 900 andthe Past time limit can be highlighted in red (or other colour codings)on the time slider 910. As the Past Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

Future Time Range

The Future Time Range selector 916 sets the range of time after themoment of interest 900 for which events will be shown. The Future Timerange is adjusted by dragging the selector 916 left and right with themouse pointer. The range between the moment of interest 900 and theFuture time limit is highlighted in blue (or other colour codings) onthe time slider 910. As the Future Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

The time range visible in the time scale of the time slider 910 can beexpanded or contracted to show a time span from centuries to seconds.Clicking and dragging on the time slider 910 anywhere except the threeselectors 912, 914, 916 will allow the entire time scale to slide totranslate in time to a point further in the future or past. Othercontrols 918 associated with the time slider 910 can be such as a “Fit”button 919 for automatically adjusting the time scale to fit the rangeof time covered by the currently active data set displayed in thevisualization representation 18. Controls 918 can include a Fit controlbutton 919, a scale-expand-contract controls 920, a step control 923,and a play control 922, which allow the user to expand or contract thetime scale. The step control 923 increments the instant of focus 900forward or back. The“playback” button 920 causes the instant of focus900 to animate forward by a user-adjustable rate. This “playback” causesthe visualization representation 18 as displayed to animate in sync withthe time slider 910.

Simultaneous Spatial and Temporal Navigation can be provided by the tool12 using, for example, interactions such as zoom-box selection and savedviews. In addition, simultaneous spatial and temporal zooming can beused to provide the user to quickly move to a context of interest. Inany view of the representation 18, the user may select a subset ofevents 20 and zoom to them in both time 402 and space 400 domains usingFit Time and a Fit Space functions through the button 919. Thesefunctions can happen simultaneously by dragging a zoom-box on to thetime chart 430 itself. The time range and the geographic extents of theselected events 20 can be used to set the bounds of the new view of therepresentation 18, including selected domain 400,402 view formats.

Referring again to FIGS. 13 and 27, the Fit control 919 of the timerslider and other controls 306 can be further subdivided into separatefit time 760 and fit geography/space 762 functions as performed by a fitmodule 700 though user interaction via the fit control button 919. Forexample, with a single click via the controls 306, for the fit togeography function the fit module 700 can instruct the visualizationmanager 300 to zoom in to user selected objects 20,21,22,23,24 (i.e.visual elements 410) and/or connection elements 412 (see FIG. 17) inboth/either space (FG) and/or time (FT), as displayed in a re-rendered“fit” version of the representation 18. For example, for fit togeography function 762, after the user has selected places, targetsand/or events (i.e. elements 410,412) from the representation 18, thefit module 700 instructs the visualization manager 300 to reduce/expandthe displayed map of the representation 18 to only the geographic areathat includes those selected elements 410,412. If nothing is selected,the map is fitted to the entire data set (i.e. all geographic areas)included in the representation 18. For example, for fit to time function760, after the user has selected places, targets and/or events (i.e.elements 410,412) from the representation 18, the fit module 700instructs the visualization manager 300 to reduce/expand the pastportion of the timeline(s) 422 to encompass only the period thatincludes the selected visual elements 410,412. Further, the fit module700 can instruct the visualization manager 300 to adjust the display ofthe browse time slider as moved to the end of the period containing theselected visual elements 410,412 and the future portion of the timeline422 can account for the same proportion of the visible timeline 422 asit did before the timeline(s) 422 were “time fitted”. If nothing isselected, the timeline is fitted to the entire data set (i.e. alltemporal areas) included in the representation 18. Further, it isrecognized, for both Fit to Geography 762 and Fit to Time 760, if onlytargets are selected, the fit module 700 coordinates the display of themap/timeline to fit to the targets' entire set of events. Further forexample, if a target is selected in addition to events, only thoseevents selected are used in the fit calculation of the fit module 700.

Referring to FIGS. 13 a and 27, operation of the fit buttons 919 of thefit module 700 begin when the user selects 768 desired objects 14(contained in the tables 122 of the memory 102 for example—see FIG. 2),such that the selected objects have time and/or geographical attributes.At step 770, the user selects the desired fit function(s) 760,762. Atstep 772, the fit module 700 analyses the range of time and/or geographyassociated with the selected object(s) 14 (i.e. maximum and minimumvalues for geography and/or time) and determines the appropriatetime/spatial range(s) in the domains 402,400 for display by thevisualization representation 18. At step 774, the fit module 700instruction the visualization manager 300 to adjust the display of thevisualization representation 18 to match the determined time/spatialranges. At step 776, the visualization manager 300 adjusts the displayeddomain 400 to match the determined spatial range and adjusts thedisplayed temporal domain 402 to math the determined temporal range, aswell as updating the indicated ranges 914,916 and the instant of focus912 of the time slider 910 (see FIG. 13).

Association Analysis Tools

Referring to FIGS. 1 and 3, an association analysis module 307 hasfunctions that have been developed that take advantage of theassociation-based connections between Events, Entities and Locations.These functions 307 are used to find groups of connected objects 14during analysis of the data in the visualization representation 18. Theassociations 16 connect these basic objects 20, 22, 24 into complexgroups 27 (see FIGS. 6 and 7) representing actual occurrences. Thefunctions are used to follow the associations 16 from object 14 toobject 14 to reveal connections between objects 14 that are notimmediately apparent. Association analysis functions are especiallyuseful in analysis of large data sets where an efficient method to findand/or filter connected groups is desirable, i.e. identify selectedobjects 14 and their associated group(s) of objects 14 from the largerscale of unconnected objects 14 in the visualization representation 18.For example, an Entity 24 maybe be involved in events 20 in a dozenplaces/locations 22, and each of those events 20 may involve otherEntities 24. The association analysis function 307 can be used todisplay only those locations 22 on the visualization representation 18that the entity 24 has visited and/or the other entities 24 that havebeen contacted.

The analysis functions A,B,C,D provide the user with different types oflink analysis that display connections between 14 of interest, such asbut limited to:

1. Expanding Search Module A, e.g. a Link Analysis Tool.

-   -   Referring to FIGS. 3, 27, and 32, the expanding search module A        of the module 307 allows the user to start with a selected        object(s) 14 a initially shown amongst other objects 14 (both        connected and not connected) in the visualization representation        18—see FIG. 32 a. The user selects the object 14 a or group of        objects 14 a of focus and clicks on the Expanding search button        920 for a degree of separation 781 of “0”, which causes        everything in the visualization representation 18 to disappear        (or to be otherwise visually distinguished from the other        objects 14—e.g. level of transparency, colour, etc.) except the        selected items, see FIG. 32 b. The user then increments the        search depth (e.g. via an appropriate depth slider control) and        objects 14 connected by the specified depth, i.e. degree of        separation 781, are made visible in the display. For example, in        FIG. 32 c the selected degree of separation 781 is “1” and all        objects 14 connected by one degree 782 are shown in the        representation 18 along with the selected object 14 a. In FIG.        32 d the selected degree of separation 781 is “2” and all        objects 14 connected by one degree 782 and two degrees 784 are        shown in the representation 18 along with the selected object 14        a.

In this way, sets of connected objects 14 are revealed as displayedusing the visual elements 410 and 412 in the representation 18 byincrementally showing objects 14 that are associated with the selectedobject 14 a by increasing degrees of separation. Accordingly, theexpanding search module A of the module 307 displays all objects 14represented by elements 410 and 412, in the representation 18 that areconnected to the selected object 14 a, within the specified range ofseparation. The range of separation of the expanding search module A canbe selected by the user using the I/O interface 108, using a linksslider 730 in a dialog window (see FIG. 31 a). For example, this linkanalysis can be performed when a single place 22, target 24 or event 20is first selected as object 14 a. An example operation of the depthslider 730 is as follows, when the expanding search module A is firstselected via the I/O interface 108, a dialog opens, and the links slider730 is initially set to 0 and only the selected object 14 a is displayedin the representation 18. Using the slider 730 (or entry field), whenthe links slider 730 is moved to 1, any object 14 directly linked (i.e.1 degree of separation such as all elementary events 20) to theinitially selected object 14 a appears on the representation 18 inaddition to the initially selected object 14 a. As the links slider 730is positioned higher up the slider scale, additional connected objects14 are added at each level to the representation 18, until all objects14 connected to the initially selected object 14 a are displayed.

2. Connection Search B, e.g. a Join Analysis Tool.

Referring to FIGS. 3, 27, and 33, the Connection Search function B ofthe module 307 allows the user to connect any pair of objects X1,X2 bytheir web of associations 26. The user selects any two target objectsX1,X2 and clicks on the Connection Search module B. The connectionsearch module B works by automatically scanning the extents of the webof associations 26 starting from one of the initially selected targetobjects X1,X2 of the pair. The search will continue until the secondtarget object X1,X2 is found as one of the connected target objectsX1,X2 or until there are no more connected target objects X1,X2. If apath 788 of associated objects 14 between the target objects X1,X2exists, all of the associated objects 14 along that path 788 aredisplayed and the depth 732 is automatically displayed showing theminimum number of links 788 between the target objects X1,X2. Forexample, FIG. 33 a shows one connection path, i.e. a depth 732 of “1”.For example, FIG. 33 b shows two connection paths, i.e. a depth 732 of“2”, with an interconnected object 14. For example, FIG. 33 c showsthree connection paths 788, i.e. a depth 732 of “3”, with twointerconnected objects 14. It is recognized that other configurations ofpaths 788 and associated objects 14 than shown can be generated by theconnection module B, in relation to the selected target objects X1,X2.

Accordingly, the Join/connection Analysis module B looks for anddisplays any specified connection path 788 between two (or more)selected target objects X1,X2. This join analysis is performed when twotarget objects X1,X2, for example, are selected from the representation18. It is noted that if the two selected target objects X1,X2 are notconnected, no events 20 are displayed and the connection level 732 isset to zero. If the paired target objects X1,X2 are connected, theshortest path 788 between them is automatically displayed, for example.It is noted that the Join Analysis module B can be generalized for threeor more selected target objects X1,X2 and their associated connectionpaths 788 and objects 14.

An example operation of the Join Analysis module B is a selection of thetargets 24 Alan and Rome. When the dialog opens, the number of links 732(e.g. 4—which is user adjustable—see FIG. 31 b) required to make aconnection between the two targets 24 is displayed to the user, and onlythe objects 14 involved in that connection (having 4 links) are visibleon the representation 18.

3. A Chain Analysis Tool C

Referring to FIGS. 3, 27, and 34, The Chain Analysis module C displaysdirect and/or indirect connections 412 between a selected target X1 andother connected targets X2. For example, in a direct connection, asingle event 20 (example of connection 412) connects target X1 andtarget X2 (who are both on the terrain 400), see FIG. 34 a. In anindirect connection, some number of events 20 (chain) connect X1 and X2,via a target X3 (who can be located off the displayed terrain 400 forexample), see FIG. 34 b. This analysis can be performed with a singleinitial target X1 selected. For example, the Analysis module C can beassociated with a chaining slider 736—see FIG. 31 c (accessed via theI/O interface 108) with the selections of such as but not limited todirect 790, indirect 792, and both 794.

For example, the target TOM is first selected on the representation 18and then when the target chaining slider 736 is set to Direct, thetargets ALAN and PARENTS are displayed, along with the events 20 thatcause TOM to be directly connected to them. In the case where TOM doesnot have any indirect target connections, so moving the slider 736 toboth 794 and to indirect 792 does not change the view as generated onthe representation 18 for the direct chaining slider 736 setting.

4. A Move Analysis Tool D

Referring to FIGS. 3, 27, and 5, this analysis module D finds, for asingle target 410, all sets of consecutive events 20 (represented byother elements 410), that are located at different places that happenedwithin the specific time range of the temporal domain 402. For example,this analysis of the analysis module D may be performed with a singletarget 410 selected from the representation 18. In example operation ofthe analysis module D, the initial target 410 is selected, when a slider736 opens (see FIG. 31 d), the time range slider 736 is set to one Yearand quite a few connected events 20 may be displayed on therepresentation 18, which are connected to the initially selected target410. When the slider 736 selection is changed to the unit type of oneWeek, the number of events 20 displayed will drop accordingly.Similarly, as the time range slider 736 is positioned higher, the numberof events 20 connected to the selected target 410 are added to therepresentation 18 as the time range increases. It is recognized that theselected target 410 can be an entity 24, and event 20, and/or a location22, as desired. It is further recognized that the selected target 410can be a group of objects 14, as desired.

It is recognized that the functions of the module 307 can be used toimplement filtering via such as but not limited to criteria matching,algorithmic methods and/or manual selection of objects 14 andassociations 16 using the analytical properties of the tool 12. Thisfiltering can be used to highlight/hide/show (exclusively) selectedobjects 14 and associations 16 as represented on the visualrepresentation 18. The functions are used to create a group (subset) ofthe objects 14 and associations 16 as desired by the user through thespecified criteria matching, algorithmic methods and/or manualselection. Further, it is recognized that the selected group of objects14 and associations 16 could be assigned a specific name which is storedin the table 122. Further, it is recognized that the module 307 andassociated analysis modules A,B,C,D can have an access module, whichaccesses the tables 122 of the memory 102 for retrieving object 14and/or association 26 data representing the links/connections determinedthrough the above described example operation of the analysis modulesA,B,C,D.

Operation of Visual Tool to Generate Visualization Representation

Referring to FIG. 14, example operation 1400 shows communications 1402and movement events 1404 (connection visual elements 412—see FIGS. 6 and7) between Entities “X” and “Y” over time on the visualizationrepresentation 18. This FIG. 14 shows a static view of Entity X makingthree phone call communications 1402 to Entity Y from 3 differentlocations 410 a at three different times. Further, the movement events1404 are shown on the visualization representation 18 indicating thatthe entity X was at three different locations 410 a (location A,B,C),which each have associated timelines 422. The timelines 422 indicate bythe relative distance (between the elements 410 b and 410 a) of theevents (E1,E2,E3) from the instant of focus 900 of the reference surface404 that these communications 1404 occurred at different times in thetime dimension 432 of the temporal domain 402. Arrows on thecommunications 1402 indicate the direction of the communications 1402,i.e. from entity X to entity Y. Entity Y is shown as remaining at onelocation 410 a (D) and receiving the communications 1402 at thedifferent times on the same timeline 422.

Referring to FIG. 15, example operation 1500 for shows Events 140 boccurring within a process diagram space domain 400 over the timedimension 432 on the reference surface 404. The spatial domain 400represents nodes 1502 of a process. This FIG. 14 shows how a flowchartor other graphic process can be used as a spatial context for analysis.In this case, the object (entity) X has been tracked through theproduction process to the final stage, such that the movements 1504represent spatial connection elements 412 (see FIGS. 6 and 7).

Referring to FIGS. 3 and 19, operation 800 of the tool 12 begins by themanager 300 assembling 802 the group of objects 14 from the tables 122via the data manager 114. The selected objects 14 are combined 804 viathe associations 16, including assigning the connection visual element412 (see FIGS. 6 and 7) for the visual representation 18 betweenselected paired visual elements 410 corresponding to the selectedcorrespondingly paired data elements 14 of the group. The connectionvisual element 412 represents a distributed association 16 in at leastone of the domains 400, 402 between the two or more paired visualelements 410. For example, the connection element 412 can representmovement of the entity object 24 between locations 22 of interest on thereference surface 404, communications (money transfer, telephone call,email, etc. . . . ) between entities 24 different locations 22 on thereference surface 404 or between entities 24 at the same location 22, orrelationships (e.g. personal, organizational) between entities 24 at thesame or different locations 22.

Next, the manager 300 uses the visualization components 308 (e.g.sprites) to generate 806 the spatial domain 400 of the visualrepresentation 18 to couple the visual elements 410 and 412 in thespatial reference frame at various respective locations 22 of interestof the reference surface 404. The manager 300 then uses the appropriatevisualization components 308 to generate 808 the temporal domain 402 inthe visual representation 18 to include various timelines 422 associatedwith each of the locations 22 of interest, such that the timelines 422all follow the common temporal reference frame. The manager 112 thentakes the input of all visual elements 410, 412 from the components 308and renders them 810 to the display of the user interface 202. Themanager 112 is also responsible for receiving 812 feedback from the uservia user events 109 as described above and then coordinating 814 withthe manager 300 and components 308 to change existing and/or create (viasteps 806, 808) new visual elements 410, 412 to correspond to the userevents 109. The modified/new visual elements 410, 412 are then renderedto the display at step 810.

Referring to FIG. 16, an example operation 1600 shows animating entity Xmovement between events (Event 1 and Event 2) during time slider 901interactions via the selector 912. First, the Entity X is observed atLocation A at time t. As the slider selector 912 is moved to the right,at time t+1 the Entity X is shown moving between known locations (Event1and Event2). It should be noted that the focus 900 of the referencesurface 404 changes such that the events 1 and 2 move along theirrespective timelines 422, such that Event 1 moves from the future intothe past of the temporal domain 402 (from above to below the referencesurface 404). The length of the timeline 422 for Event 2 (between theEvent 2 and the location B on the reference surface 404 decreasesaccordingly. As the slider selector 912 is moved further to the right,at time t+2, Entity X is rendered at Event2 (Location B). It should benoted that the Event 1 has moved along its respective timeline 422further into the past of the temporal domain 402, and event 2 has movedaccordingly from the future into the past of the temporal domain 402(from above to below the reference surface 404), since therepresentation of the events 1 and 2 are linked in the temporal domain402. Likewise, the entity X is linked spatially in the spatial domain400 between event 1 at location A and event 2 at location B. It is alsonoted that the Time Slider selector 912 could be dragged along the timeslider 910 by the user to replay the sequence of events from time t tot+2, or from t+2 to t, as desired.

Referring to FIGS. 27 and 35, a further feature of the tool 12 is atarget tracing module 722, which takes user input from the I/O interface108 for tracing of a selected target/entity 24 through associated events20 a,b,c,d. For example, the user of the tool 12 selects one of theevents 20 a from the representation 18 associated with one or moreentities/target 24, whereby the module 722 provides for a selection icon796 to be displayed adjacent to the selected event 20 a on therepresentation 18. Using the interface 108 (e.g. up/down arrows), theuser can navigate the representation 18 by scrolling, represented byreference numeral 798 and example vector) back and forward (in terms oftime and/or geography) through the events 20 b,c,d associated with thattarget 24, i.e. the display of the representation 18 adapts as the usersequentially scrolls 798, for example, through the time domain 402, asdescribed already above. For example, the display of the representation18 moves between consecutive events 20 a,b,c,d associated with thetarget 24.

Referring to FIG. 35, in an example implementation of the I/O interface08, the Page Up key moves the selection icon 796 upwards (back in time)and the Page Down key moves the selection icon 796 downwards (forward intime), such that after selection of a single event 20 a with anassociated target 24, the Page Up keyboard key would move the selectionicon 796 to the next event 20 b (back in time) on the associatedtarget's 24 trail while selecting the Page Down key would return theselection icon 796 to the first event 20 a selected. The module 722coordinates placement of the selection icon 796 at consecutive events 20b,c,d connected with the associated target 24 while skipping over thoseevents 20 (while scrolling) not connected with the associated target 24.

Referring to FIG. 17, the visual representation 18 shows connectionvisual elements 412 between visual elements 410 situated on selectedvarious timelines 422. The timelines 422 are coupled to variouslocations 22 of interest on the geographical reference frame 404. Inthis case, the elements 412 represent geographical movement betweenvarious locations 22 by entity 24, such that all travel happened at sometime in the future with respect to the instant of focus represented bythe reference plane 404.

Referring to FIG. 18, the spatial domain 400 is shown as a geographicalrelief map. The timechart 430 is superimposed over the spatial domain ofthe visual representation 18, and shows a time period spanning fromDecember 3^(rd) to January 1^(st) for various events 20 and entities 24situated along various timelines 422 coupled to selected locations 22 ofinterest. It is noted that in this case the user can use the presentedvisual representation to coordinate the assignment of various connectionelements 412 to the visual elements 410 (see FIG. 6) of the objects 20,22, 24 via the user interface 202 (see FIG. 1), based on analysis of thedisplayed visual representation 18 content. A time selection 950 isJanuary 30, such that events 20 and entities 24 within the selection boxcan be further analysed. It is recognised that the time selection 950could be used to represent the instant of focus 900 (see FIG. 9).

Aggregation Module 600

Referring to FIG. 3, an Aggregation Module 600 is for, such as but notlimited to, summarizing or aggregating the data objects 14, providingthe summarized or aggregated data objects 14 to the VisualizationManager 300 which processes the translation from data objects 14 andgroup of data elements 27 to the visual representation 18, and providingthe creation of summary charts 200 (see FIG. 26) for displayinginformation related to summarised/aggregated data objects 14 as thevisual representation 18 on the display 108.

Referring to FIGS. 3 and 22, the spatial inter-connectedness ofinformation over time and geography within a single, highly interactive3-D view of the representation 18 is beneficial to data analysis (of thetables 122). However, when the number of data objects 14 increases,techniques for aggregation become more important. Many individuallocations 22 and events 20 can be combined into a respective summary oraggregated output 603. Such outputs 603 of a plurality of individualevents 20 and locations 22 (for example) can help make trends in timeand space domains 400,402 more visible and comparable to the user of thetool 12. Several techniques can be implemented to support aggregation ofdata objects 14 such as but not limited to techniques of hierarchy oflocations, user defined geo-relations, and automatic LOD levelselection, as further described below. The tool 12 combines the spatialand temporal domains 400, 402 on the display 108 for analysis of complexpast and future events within a selected spatial (e.g. geographic)context.

Referring to FIG. 22, the Aggregation Module 600 has an AggregationManager 601 that communicates with the Visualization Manager 300 forreceiving aggregation parameters used to formulate the output 603. Theparameters can be either automatic (e.g. tool pre-definitions) manual(entered via events 109) or a combination thereof. The manager 601accesses all possible data objects 14 through the Data Manager 114(related to the aggregation parameters—e.g. time and/or spatial rangesand/or object 14 types/combinations) from the tables 122, and thenapplies aggregation tools or filters 602 for generating the output 603.The Visualization Manager 300 receives the output 603 from theAggregation Manager 601, based on the user events 109 and/or operationof the Time Slider and other Controls 306 by the user for providing theaggregation parameters. As described above, once the output 603 isrequested by the Visualization Manager 114, the Aggregation Manager 601communicates with the Data Manager 114 access all possible data objects14 for satisfying the most general of the aggregation parameters andthen applies the filters 602 to generate the output 603. It isrecognised however, that the filters 602 could be used by the manager601 to access only those data objects 14 from the tables 122 thatsatisfy the aggregation parameters, and then copy those selected dataobjects 14 from the tables 122 for storing/mapping as the output 603.

Accordingly, the Aggregation Manager 601 can make available the dataelements 14 to the Filters 602. The filters 602 act to organize andaggregate (such as but not limited to selection of data objects 14 fromthe global set of data in the tables 122 according to rules/selectioncriteria associated with the aggregation parameters) the data objects 14according the instructions provided by the Aggregation Manager 601. Forexample, the Aggregation Manager 601 could request that the Filters 602summarize all data objects 14 with location data 22 corresponding toParis. Or, in another example, the Aggregation Manager 601 could requestthat the Filters 602 summarize all data objects 14 with event data 20corresponding to Wednesdays. Once the data objects 14 are selected bythe Filters 602, the aggregated data is summarised as the output 603.The Aggregation Manager 601 then communicates the output 603 to theVisualization Manager 300, which processes the translation from theselected data objects 14 (of the aggregated output 603) for rendering asthe visual representation 18. It is recognised that the content of therepresentation 18 is modified to display the output 603 to the user ofthe tool 12, according to the aggregation parameters.

Further, the Aggregation Manager 601 provides the aggregated dataobjects 14 of the output 603 to a Chart Manager 604. The Chart Manager604 compiles the data in accordance with the commands it receives fromthe Aggregation Manager 601 and then provides the formatted data to aChart Output 605. The Chart Output 605 provides for storage of theaggregated data in a Chart section 606 of the display (see FIG. 25).Data from the Chart Output 605 can then be sent directly to theVisualization Renderer 112 or to the visualisation manager 300 forinclusion in the visual representation 18, as further described below.

Referring to FIG. 23, an example aggregation of data objects 14 by theAggregation Module 601 is shown. The event data 20 (for example) isaggregated according to spatial proximity (threshold) of the dataobjects 14 with respect to a common point (e.g. particular location 410or other newly specified point of the spatial domain 400), differencethreshold between two adjacent locations 410, or other spatial criteriaas desired. For example, as depicted in FIG. 23 a, the three dataobjects 20 at three locations 410 are aggregated to two objects 20 atone location 410 and one object at another location 410 (e.g.combination of two locations 410) as a user-defined field 202 of view isreduced in FIG. 23 b, and ultimately to one location 410 with all threeobjects 20 in FIG. 23 c. It is recognised in this example of aggregatedoutput 603 that timelines 422 of the locations 410 are combined asdictated by the aggregation of locations 410.

For example, the user may desire to view an aggregate of data objects 14related within a set distance of a fixed location, e.g., aggregate ofevents 20 occurring within 50 km of the Golden Gate Bridge. Toaccomplish this, the user inputs their desire to aggregate the dataaccording to spatial proximity, by use of the controls 306, indicatingthe specific aggregation parameters. The Visualization Manager 300communicates these aggregation parameters to the Aggregation Module 600,in order for filtering of the data content of the representation 18shown on the display 108. The Aggregation Module 600 uses the Filters602 to filter the selected data from the tables 122 based on theproximity comparison between the locations 410. In another example, ahierarchy of locations can be implemented by reference to theassociation data 26 which can be used to define parent-childrelationships between data objects 14 related to specific locationswithin the representation 18. The parent-child relationships can be usedto define superior and subordinate locations that determine the level ofaggregation of the output 603.

Referring to FIG. 24, an example aggregation of data objects 14 by theAggregation Module 601 is shown. The data 14 is aggregated according todefined spatial boundaries 204. To accomplish this, the user inputstheir desire to aggregate the data 14 according to specific spatialboundaries 204, by use of the controls 306, indicating the specificaggregation parameters of the filtering 602. For example, a user maywish to aggregate all event 20 objects located within the city limits ofToronto. The Visualization Manager 300 then requests to the AggregationModule 600 to filter the data objects 14 of the current representationaccording to the aggregation parameters. The Aggregation Module 600provides implements or otherwise applies the filters 602 to filter thedata based on a comparison between the location data objects 14 and thecity limits of Toronto, for generating the aggregated output 603. InFIG. 24 a, within the spatial domain 205 the user has specified tworegions of interest 204, each containing two locations 410 withassociated data objects 14. In FIG. 24 b, once filtering has beenapplied, the locations 410 of each region 204 have been combined suchthat now two locations 410 are shown with each having the aggregatedresult (output 603) of two data objects 14 respectively. In FIG. 24 c,the user has defined the region of interest to be the entire domain 205,thereby resulting in the displayed output 603 of one location 410 withthree aggregated data objects 14 (as compared to FIG. 24 a). It is notedthat the positioning of the aggregated location 410 is at the center ofthe regions of interest 204, however other positioning can be used suchas but not limited to spatial averaging of two or more locations 410 orplacing aggregated object data 14 at one of the retained originallocations 410, or other positioning techniques as desired.

In addition to the examples in illustrated in FIGS. 21 and 22, theaggregation of the data objects can be accomplished automatically basedon the geographic view scale provided in the visual representations.Aggregation can be based on level of detail (LOD) used in mappinggeographical features at various scales. On a 1:25,000 map, for example,individual buildings may be shown, but a 1:500,000 map may show just apoint for an entire city. The aggregation module 600 can supportautomatic LOD aggregation of objects 14 based on hierarchy, scale andgeographic region, which can be supplied as aggregation parameters aspredefined operation of the controls 306 and/or specific manualcommands/criteria via user input events 109. The module 600 can alsointeract with the user of the tool 12 (via events 109) to adjust LODbehaviour to suit the particular analytical task at hand.

Referring to FIG. 27 and FIG. 28, the aggregation module 600 can alsohave a place aggregation module 702 for assigning visual elements410,412 (e.g. events 20) of several places/locations 22 to one commonaggregation location 704, for the purpose of analyzing data for anentire area (e.g. a convoy route or a county). It is recognised that theplace aggregation function can be turned on and off for each aggregationlocation 704, so that the user of the tool 12 can analyze data with andwithout the aggregation(s) active. For example, the user creates theaggregation location 704 in a selected location of the spatial domain400 of the representation 18. The user then gives the createdaggregation location 704 a label 706 (e.g. North America). The user thenselects a plurality of locations 22 from the representation, eitherindividually or as a group using a drawing tool 707 to draw around alldesired locations 22 within a user defined region 708. Once selected,the user can drag or toggle the selected regions 708 and individuallocations 22 to be included in the created aggregation location 704 bythe aggregation module 702. The aggregation module 702 could instructthe visualization manager 300 to refresh the display of therepresentation 18 to display all selected locations 22 and relatedvisual elements 410,412 in the created aggregation location 704. It isrecognised that the aggregation module 702 could be used to configurethe created aggregation location 704 to display other selected objecttypes (e.g. entities 24) as a displayed group. In the case of selectedentities 24, the created aggregation location 704 could be labelled theselected entities' name and all visual elements 410,412 associated withthe selected entity (or entities) would be displayed in the createdaggregation location 704 by the aggregation module 702. It is recognisedthat the above-described same aggregation operation could be done forselected event 20 types, as desired.

Referring to FIG. 25, an example of a spatial and temporal visualrepresentation 18 with summary chart 200 depicting event data 20 isshown. For example, a user may wish to see the quantitative informationrelating to a specific event object. The user would request the creationof the chart 200 using the controls 306, which would submit the requestto the Visualization Manager 300. The Visualization Manager 300 wouldcommunicate with the Aggregation Module 600 and instruct the creation ofthe chart 200 depicting all of the quantitative information associatedwith the data objects 14 associated with the specific event object 20,and represent that on the display 108 (see FIG. 2) as content of therepresentation 18. The Aggregation Module 600 would communicate with theChart Manager 604, which would list the relevant data and provide onlythe relevant information to the Chart Output 605. The Chart Output 605provides a copy of the relevant data for storage in the Chart ComparisonModule, and the data output is communicated from the Chart Output 605 tothe Visualization Renderer 112 before being included in the visualrepresentation 18. The output data stored in the Chart Comparisonsection 606 can be used to compare to newly created charts 200 whenrequested from the user. The comparison of data occurs by selectingparticular charts 200 from the chart section 606 for application as theoutput 603 to the Visual Representation 18.

The charts 200 rendered by the Chart Manager 604 can be created in anumber of ways. For example, all the data objects 14 from the DataManager 114 can be provided in the chart 200. Or, the Chart Manager 604can filter the data so that only the data objects 14 related to aspecific temporal range will appear in the chart 200 provided to theVisual Representation 18. Or, the Chart Manager 604 can filter the dataso that only the data objects 14 related to a specific spatial andtemporal range will appear in the chart 200 provided to the VisualRepresentation 18.

Referring to FIG. 30, a further embodiment of event aggregation charts200 calculates and displays (both visually and numerically) the countobjects by various classifications 726. When charts 200 are displayed onthe map (e.g. on-map chart), one chart 200 is created for each place 22that is associated with relevant events 20. Additional options becomeavailable by clicking on the colored chart bars 728 (e.g. Hide selectedobjects, Hide target). By default, the chart manager 604 (see FIG. 22)can assign colors to chart bars 728 randomly, except for example whenthey are for targets 24, in which case the chart manager 604 usesexisting target 24 colors, for convenience. It is noted that a Chartscale slider 730 can be used to increase or decrease the scale of on-mapcharts 200, e.g. slide right or left respectively. The chart manager 604can generate the charts 200 based on user selected options 724, such asbut not limited to:

1) Show Charts on Map—presents a visual display on the map, one chart200 for each place 22 that has relevant events 20;

2) Chart Events in Time Range Only—includes only events 20 that happenedduring the currently selected time range;

3) Exclude Hidden Events—excludes events 20 that are not currentlyvisible on the display (occur within current time range, but arehidden);

4) Color by Event—when this option is turned on, event 20 color is usedfor any bar 728 that contains only events 20 of that one color. When abar 728 contains events 20 of more than one color, it is displayed gray;

5) Sort by Value—when turned on, results are displayed in the Charts 200panel, sorted by their value, rather than alphabetically; and

6) Show Advanced Options—gives access to additional statisticalcalculations.

In a further example of the aggregation module 601, user-definedlocation boundaries 204 can provide for aggregation of data 14 across anarbitrary region. Referring to FIG. 26, to compare a summary of eventsalong two separate routes 210 and 212, aggregation output 603 of thedata 14 associated with each route 210,212 would be created by drawingan outline boundary 204 around each route 210,212 and then assigning theboundaries 204 to the respective locations 410 contained therein, asdepicted in FIG. 26 a. By the user adjusting the aggregation level inthe Filters 602 through specification of the aggregation parameters ofthe boundaries 204 and associated locations 410, the data 14 is theaggregated as output 603 (see FIG. 26 b) within the outline regions intothe newly created locations 410, with the optional display of text 214providing analysis details for those new aggregated locations 410. Forexample, the text 214 could summarise that the number of bad events 20(e.g. bombings) is greater for route 210 than route 212 and thereforeroute 212 would be the route of choice based on the aggregated output603 displayed on the representation 18.

It will be appreciated that variations of some elements are possible toadapt the invention for specific conditions or functions. The conceptsof the present invention can be further extended to a variety of otherapplications that are clearly within the scope of this invention.

For example, one application of the tool 12 is in criminal analysis bythe “information producer”. An investigator, such as a police officer,could use the tool 12 to review an interactive log of events 20 gatheredduring the course of long-term investigations. Existing reports andquery results can be combined with user input data 109, assertions andhypotheses, for example using the annotations 21. The investigator canreplay events 20 and understand relationships between multiple suspects,movements and the events 20. Patterns of travel, communications andother types of events 20 can be analysed through viewing of therepresentation 18 of the data in the tables 122 to reveal such as butnot limited to repetition, regularity, and bursts or pauses in activity.

Subjective evaluations and operator trials with four subject matterexperts have been conducted using the tool 12. These initial evaluationsof the tool 12 were run against databases of simulated battlefieldevents and analyst training scenarios, with many hundreds of events 20.These informal evaluations show that the following types of informationcan be revealed and summarised. What significant events happened in thisarea in the last X days? Who was involved? What is the history of thisperson? How are they connected with other people? Where are the activityhot spots? Has this type of event occurred here or elsewhere in the lastY period of time?

With respect to potential applications and the utility of the tool 12,encouraging and positive remarks were provided by military subjectmatter experts in stability and support operations. A number of thoseremarks are provided here. Preparation for patrolling involvedresearching issues including who, where and what. The history of localbelligerent commanders and incidents. Tracking and being aware ofhistory, for example, a ceasefire was organized around a religiouscalendar event. The event presented an opportunity and knowing about theevent made it possible. In one campaign, the head of civil affairs hadbeen there twenty months and had detailed appreciation of the historyand relationships. Keeping track of trends. What happened here? Whatkeeps happening here? There are patterns. Belligerents keep trying thesame thing with new rotations [a rotation is typically six to twelvemonths tour of duty]. When the attack came, it did come from the areawhere many previous earlier attacks had also originated. The discoveryof emergent trends . . . persistent patterns . . . sooner rather thanlater could be useful. For example, the XXX Colonel that tends to showup in an area the day before something happens. For every rotation avaluable knowledge base can be created, and for every rotation, thisknowledge base can be retained using the tool 12 to make the knowledgebase a valuable historical record. The historical record can includeevents, factions, populations, culture, etc.

Referring to FIG. 27, the tool 12 could also have a report generationmodule 720 that saves a JPG format screenshot (or other picture format),with a title and description (optional—for example entered by the user)included in the screenshot image, of the visual representation 18displayed on the visual interface 202 (see FIG. 1). For example, thescreenshot image could include all displayed visual elements 410,412,including any annotations 21 or other user generated analysis related tothe displayed visual representation 18, as selected or otherwisespecified by the user. A default mode could be all currently displayedinformation is captured by the report generation module 720 and saved inthe screenshot image, along with the identifying label (e.g. titleand/or description as noted above) incorporated as part of thescreenshot image (e.g. superimposed on the lower right-hand corner ofthe image). Otherwise the user could select (e.g. from a menu) whichsubset of the displayed visual elements 410,412 (on acategory/individual basis) is for inclusion by the module 720 in thescreenshot image, whereby all non-selected visual elements 410,412 wouldnot be included in the saved screenshot image. The screenshot imagewould then be given to the data manager 114 (see FIG. 3) for storing inthe database 122. For further information detail of the visualrepresentation 18 not captured in the screenshot image, a filename (orother link such as a URL) to the non-displayed information could also besuperimposed on the screenshot image, as desired. Accordingly, the savedscreenshot image can be subsequently retrieved and used as a quickvisual reference for more detailed underlying analysis linked to thescreenshot image. Further, the link to the associated detailed analysiscould be represented on the subsequently displayed screenshot image as ahyperlink to the associated detailed analysis, as desired.

Having thus described the present invention with respect to preferredembodiments as implemented, it will be apparent to those skilled in theart that many modifications and enhancements are possible to the presentinvention without departing from the basic concepts as described in thepreferred embodiment of the present invention. Therefore, what isintended to be protected by way of letters patent should be limited onlyby the scope of the following claims.

1. A method for analysing a plurality of data elements having bothtemporal and spatial properties, a first data element and a second dataelement of the plurality of data elements linked by at least oneassociation element, the method comprising the steps of: selecting thefirst data element from the plurality of data elements; providing atleast one search criteria for use in analysing the properties of theplurality of data elements with respect to at least one property of thefirst data element; applying the at least one search criteria to theproperties of the plurality of data elements for identifying the seconddata element from the plurality of data elements and the correspondingat least one association element, the at least one association elementconfigured for representing a connection between the first data elementand the second data element, the connection having a first propertycommon to a property of the first data element and a second propertycommon to a property of the second data element; and generating a visualrepresentation of the first and second data elements and the associationelement configured for display on a user interface for subsequentinteraction with user events, the visual representation including aspatial domain including a reference surface for providing a spatialreference frame having at least two spatial dimensions and including atemporal domain operatively coupled to the spatial domain for providinga common temporal reference frame for locations of interest in thespatial domain.
 2. The method of claim 1, wherein the visualrepresentation of the group of data elements is selected from the groupcomprising; a concurrent time and geographic context and a concurrenttime and diagrammatic context.
 3. The method of claim 2, wherein thetype of the data elements is selected from the group comprising:entities; events; and locations.
 4. The method of claim 3, wherein theevent type represents an action taking place at a particular one of thelocations of interest in the spatial reference frame and at a particulartime in the temporal domain.
 5. The method of claim 3, wherein theentity type represents an actor involved in a selected event.
 6. Themethod of claim 3, wherein the location type represents one of helocations of interest in the spatial domain.
 7. The method of claim 2,wherein the association element is selected from the group comprising: acommunication connection describing communication details transferredbetween the two or more data elements; a financial transaction or othertransaction; and a relationship connection describing social details incommon between the two or more data elements.
 8. The method of claim 7,wherein the association element is selected from the group comprising: adirect connection; and an indirect connection.
 9. The method of claim 2further comprising the step of displaying an information element on thevisual representation adjacent to one of the elements in response to atrigger event, the information element for describing selected ones ofthe properties of said one of the elements.
 10. The method of claim 9,wherein the trigger event is a mouse over of said one of the elements.11. The method of claim 9, wherein the information element is configuredbased on an element type of said one of the elements.
 12. The method ofclaim 2 further comprising the step of adjusting the display of thevisual representation to match a selected range in at least one of thedomains.
 13. The method of claim 12 further comprising the step ofselecting the range based on the properties of the first and second dataelements.
 14. The method of claim 2, wherein the at least one searchcriteria represents a degree of separation between the first dataelement and the second data element.
 15. The method of claim 14 furthercomprising the step of selecting the degree of separation greater thanone for facilitating the identification of at least a third data elementand an additional association element, such that the association elementlinks the first data element to the second data element and theadditional association element links the second data element to thethird data element for representing a link between the first and thirddata elements.
 16. The method of claim 15 further comprising the step ofincreasing the degree of separation for resulting in the identificationand display of further data elements indirectly linked to the first dataelement through further association elements.
 17. The method of claim 2further comprising the step of selecting the second data element andapplying the at least one search criteria as a number of connections tothe properties of the second data element for facilitatingidentification of a connection path represented by the associationelement between the first and second data elements.
 18. The method ofclaim 17 further comprising the step of increasing the specified numberof connections between the first and second data elements forfacilitating identification of multiple connection paths between thefirst and second data elements and at least one additional data element.19. The method of claim 17, wherein the connection path is selected fromthe group comprising: a direct connection path and an indirectconnection path.
 20. The method of claim 15, wherein the first dataelement represents a target object and the second data elementrepresents an event connected to the target object.
 21. A system foranalysing a plurality of data elements having both temporal and spatialproperties, a first data element and a second data element of theplurality of data elements linked by at least one association element,the system comprising: a user interface for selecting the first dataelement from the plurality of data elements and for providing at leastone search criteria for use in analysing the properties of the pluralityof data elements with respect to at least one property of the first dataelement; an analysis module configured for applying the at least onesearch criteria to the properties of the plurality of data elements foridentifying the second data element from the plurality of data elementsand the corresponding at least one association element, the at least oneassociation element configured for representing a connection between thefirst data element and the second data element, the connection having afirst property common to a property of the first data element and asecond property common to a property of the second data element; and avisualisation module configured for generating a visual representationof the first and second data elements and the association elementconfigured for display on the user interface for subsequent interactionwith user events, the visual representation including a spatial domainincluding a reference surface for providing a spatial reference framehaving at least two spatial dimensions and including a temporal domainoperatively coupled to the spatial domain for providing a commontemporal reference frame for locations of interest in the spatialdomain.
 22. The system of claim 21, wherein the visual representation ofthe group of data elements is selected from the group comprising; aconcurrent time and geographic context and a concurrent time anddiagrammatic context.
 23. The system of claim 22, wherein the type ofthe data elements is selected from the group comprising: entities;events; and locations.
 24. The system of claim 23, wherein the eventtype represents an action taking place at a particular one of thelocations of interest in the spatial reference frame and at a particulartime in the temporal domain.
 25. The system of claim 23, wherein theentity type represents an actor involved in a selected event.
 26. Thesystem of claim 23, wherein the location type represents one of thelocations of interest in the spatial domain.
 27. The system of claim 22,wherein the association element is selected from the group comprising: acommunication connection describing communication details transferredbetween the two or more data elements; a financial transaction or othertransaction; and a relationship connection describing social details incommon between the two or more data elements.
 28. The system of claim27, wherein the association element is selected from the groupcomprising: a direct connection; and an indirect connection.
 29. Thesystem of claim 22 further comprising an information module configuredfor generating an information element in response to a trigger event fordisplay on the visual representation adjacent to one of the elements,the information element for describing selected ones of the propertiesof said one of the elements.
 30. The system of claim 29, wherein thetrigger event is a mouse over of said one of the elements.
 31. Thesystem of claim 29, wherein the information element is configured basedon an element type of said one of the elements.
 32. The system of claim22 further comprising a fit module for adjusting the display of thevisual representation to match a selected range in at least one of thedomains.
 33. The system of claim 32, wherein the range is based on theproperties of the first and second data elements identified by the fitmodule.
 34. The system of claim 22, wherein the at least one searchcriteria represents a degree of separation between the first dataelement and the second data element.
 35. The system of claim 14 furthercomprising the analysis module configured for applying the degree ofseparation greater than one for facilitating the identification of atleast a third data element and an additional association element, suchthat the association element links the first data element to the seconddata element and the additional association element links the seconddata element to the third data element for representing a link betweenthe first and third data elements.
 36. The system of claim 35, whereinan increase in the degree of separation results in the identificationand display of further data elements indirectly linked to the first dataelement through further association elements.
 37. The system of claim 22further comprising the user interface configured for selecting thesecond data element and the analysis module configured for applying theat least one search criteria as a number of connections to theproperties of the second data element for facilitating identification ofa connection path represented by the association element between thefirst and second data elements.
 38. The system of claim 37 further,wherein an increase in the specified number of connections between thefirst and second data elements facilitates identification of multipleconnection paths between the first and second data elements and at leastone additional data element.
 39. The system of claim 37, wherein theconnection path is selected from the group comprising: a directconnection path and an indirect connection path.
 40. The system of claim35, wherein the first data element represents a target object and thesecond data element represents an event connected to the target object.